PHP Class ElasticSource

Author: David Kullmann
Inheritance: extends DataSource
显示文件 Open project: dkullmann/cakephp-elastic-search-datasource Class Usage Examples

Public Properties

Property Type Description
$Http object Holds HttpSocket Object
$columns array Elasticsearch column definition
$fullDebug string Prevents error from being thrown in CakeTestFixture
$useNestedTransactions string Prevents error from being thrown in CakeTestFixture

Protected Properties

Property Type Description
$_document array The document for this transaction (for saving more than one model)
$_filterOps string Valid operators from DboSource to support
$_id string If we are in a transaction this is the id for the model we are indexing
$_listSources boolean Don't list sources by default, there's still some problems here
$_parent string Holding data for _parent.
$_queryLog string Query log
$_schema array Holds the mappings
$_transactionStarted boolean Track if we are in a transaction or not - allows saving multiple models to a document
$_type string If we are in a transaction this is the 'type' for the model indexing

Public Methods

Method Description
__call ( string $method, array $arguments ) : mixed Call HttpSocket methods
__construct ( array $config = [] ) Constructor, call the parent and setup Http
addToDocument ( Model $Model, array $document = [] ) : boolean Store a document to save later, merge it with existing model information for that document
afterParseConditions ( Model $Model, array $filters = [] ) : array Perform this check after parseConditions has completed, since parseConditions is recursive we have to perform this check in a separate method (or use a static variable or whatever, but, I think this is cleaner)
allDocuments ( ) : array Get all documents for this transaction
begin ( ) : true Begin a transaction - allows saving of multiple models pseudo-automically through _bulk API, but it's main purpose is to allow saving associated model data and multiple models (saveMany/saveAssociated)
bulkIndex ( string $type = null, array $documents = [] ) : boolean Bulk index a group of models
calculate ( Model $Model, $func, $params = null )
checkMapping ( Model $Model ) : boolean Check to see if a mapping exists
commit ( ) : boolean Save all models we have been storing
create ( Model $Model, array $fields = [], array $values = [] ) : mixed If this operation is atomic then store the model data to be created on "commit" later, this allows for saving multiple models in one document through saveAll, saveMany, and saveAssociated.
createIndex ( $index, $alias = false, $settings = [] )
createSchema ( $Schema ) : void Create schema, used in testing
currentDocument ( ) : array Get the current document
currentModel ( Model &$Model ) : Model Get/set the current model - called when requests are starting
delete ( Model $Model, array $conditions = null ) : boolean Delete a record
describe ( $model ) : array Describe a model based on it's mapping
dropIndex ( $index )
dropMapping ( Model $Model ) : boolean Delete a mapping
dropSchema ( string $Schema ) : void Drop a schema
execute ( $method = null, $type = null, $api = null, $data = [] )
filterResults ( array $results = [] ) : array Filter results from a call, parsing out the records
fullTableName ( Model | string $model, boolean $quote = true, boolean $schema = true ) : string Gets full table name including prefix
generateQuery ( Model $Model, array $queryData = [] ) : array Generate an ElasticSearch query from Cake's ORM $queryData
geo ( $key, $operator, $value )
geoBoundary ( $field, array $boundaries ) : array Creates filter for a geo boundary box
geoDistance ( $field, array $options ) : array Creates filter for a geo distance search
geoDistanceRange ( $field, $minDistance ) : array Creates filter for a geo distance range search
getColumnType ( Model $model, string $column ) : string Returns the column type of a column in the model.
getLog ( ) : void Get the query log - support for DebugKit Toolbar
getMapping ( ) : array Get the entire index mapping
getType ( Model $Model ) : void Get the useType if its set, otherwise use the table name
inTransaction ( ) : void Check to see if we are in a transacion
index ( $type, $id = null, $document = [] )
insertMulti ( $type, array $fields, array $values, $refresh = true ) : void Used by CakeTestFixture to insert many records.
isConnected ( ) : boolean Returns true to signal that the source is connected to the datastore
listSources ( $data = NULL ) : array List the types available for each index
logQuery ( string $method, string $uri, string $body, string $results = [] ) : void Log a new query if debug is on
mapModel ( Model $Model, array $description = [], $alias = true ) : boolean Map a model based on it's description from MySQL (or your own)
missing ( $key, $value )
name ( string $alias ) : string Make Elastic play nice with Model::escapeField();
parseConditions ( Model $Model, array $conditions = [] ) : array Parse the 'conditions' key of a query from CakePHP's ORM
parseMapping ( array $mapping = [], $sourcesOnly = false ) : array Parse an entire index mapping to create the schema for this datasource
parseOrder ( Model $Model, $query = [] )
parseQueryType ( $query )
query ( $method, $params, Model $Model )
range ( $key, $operator, $value )
read ( Model $Model, array $queryData = [], $recursive = NULL ) : mixed Query ElasticSearch to retrieve records
reindex ( ElasticScroll $scroll, array $options = [] ) : void Copies all documents from an index type to another
reset ( ) : void Reset our transaction state, document, id, and type
resetSequence ( string $table, string $column ) : boolean Reset a sequence based on the MAX() value of $column. Useful for resetting sequences after using insertMulti().
rollback ( ) : void Cancel this save operation/transaction
scan ( Model $model, integer $pageSize = 50, string $cursorTtl = '2m' ) : ElasticScroll Returns an iterator with all results for an index type, the results order is maintained by Elastic search for $cursorTtl time in minutes. This function is commonly used for re-indexing a type when it's internal pproperty definition change.
term ( $key, $operator, $value )
truncate ( string $type, $refresh = true ) : void Used by CakeTestFixture to truncate a type
typeToAlias ( $type ) : string Convert a pluralized table name (test_models) to an alias (TestModel)
update ( Model $Model, $fields = [], $values = [], $conditions = NULL )

Protected Methods

Method Description
_addLog ( string $data = [] ) : void Added an item to the query log
_convertAttributes ( string $attr, string $val ) : string Convert MySQL or CakePHP ORM field attributes into ElasticSearch compatible attributes
_findKey ( Model $Model, array $document = [] ) : mixed Find the key for this document
_normalizeBoolFilter ( array $filter ) : array Restructures bool filters so they have the propper formatting for Elastic Search
_parseDescription ( array $description = [], $Model = null ) : array Recursive method to map a SQL-like model description into a ElasticSearch one
_parseKey ( Model $Model, string $key, mixed $value ) : array Used to parse a key for ElasticSearch filters
_parseResponse ( CakeResponse $response ) : mixed Parse the response from ElasticSearch, throwing errors if necessary
_setupTransaction ( Model $Model, array $document = [] ) : void If we are in a new transaction, set _type and _id, if we are in an existing transaction then start a new document and set _id
_throwError ( string $info ) : void Throw the right error
_uri ( array $config ) : array Get the URI for a request

Method Details

__call() public method

Call HttpSocket methods
See also: ElasticSource::_parseResponse()
See also: ElasticSource::_filterResults()
Author: David Kullmann
public __call ( string $method, array $arguments ) : mixed
$method string
$arguments array
return mixed Results from HttpSocket calls parsed by _parseResponse and _filterResults

__construct() public method

Constructor, call the parent and setup Http
Author: David Kullmann
public __construct ( array $config = [] )
$config array

_addLog() protected method

Added an item to the query log
Author: David Kullmann
protected _addLog ( string $data = [] ) : void
$data string
return void

_convertAttributes() protected method

Convert MySQL or CakePHP ORM field attributes into ElasticSearch compatible attributes
Author: David Kullmann
protected _convertAttributes ( string $attr, string $val ) : string
$attr string
$val string
return string ES compatible attribute

_findKey() protected method

Find the key for this document
Author: David Kullmann
protected _findKey ( Model $Model, array $document = [] ) : mixed
$Model Model
$document array
return mixed Boolean false if no key is present, otherwise the key (string/int)

_normalizeBoolFilter() protected method

Restructures bool filters so they have the propper formatting for Elastic Search
protected _normalizeBoolFilter ( array $filter ) : array
$filter array single bool filter created by _parseKey
return array

_parseDescription() protected method

Recursive method to map a SQL-like model description into a ElasticSearch one
Author: David Kullmann
protected _parseDescription ( array $description = [], $Model = null ) : array
$description array
return array Array representing ES Mapping

_parseKey() protected method

Used to parse a key for ElasticSearch filters
Author: David Kullmann
protected _parseKey ( Model $Model, string $key, mixed $value ) : array
$Model Model
$key string
$value mixed
return array ElasticSearch compatible filter

_parseResponse() protected method

Parse the response from ElasticSearch, throwing errors if necessary
Author: David Kullmann
protected _parseResponse ( CakeResponse $response ) : mixed
$response CakeResponse
return mixed boolean true or false, or body of request as array

_setupTransaction() protected method

If we are in a new transaction, set _type and _id, if we are in an existing transaction then start a new document and set _id
Author: David Kullmann
protected _setupTransaction ( Model $Model, array $document = [] ) : void
$Model Model
$document array
return void

_throwError() protected method

Throw the right error
Author: David Kullmann
protected _throwError ( string $info ) : void
$info string
return void

_uri() protected method

Get the URI for a request
Author: David Kullmann
protected _uri ( array $config ) : array
$config array HttpSocket $config style array
return array Array compatible with HttpSocket $uri for get/post/put/delete

addToDocument() public method

Allows saveAssociated to work and allows storing multiple models in one document
Author: David Kullmann
public addToDocument ( Model $Model, array $document = [] ) : boolean
$Model Model
$document array
return boolean true

afterParseConditions() public method

Perform this check after parseConditions has completed, since parseConditions is recursive we have to perform this check in a separate method (or use a static variable or whatever, but, I think this is cleaner)
Author: David Kullmann
public afterParseConditions ( Model $Model, array $filters = [] ) : array
$Model Model
$filters array
return array

allDocuments() public method

Get all documents for this transaction
Author: David Kullmann
public allDocuments ( ) : array
return array Array of documents keyed by their primary key

begin() public method

Begin a transaction - allows saving of multiple models pseudo-automically through _bulk API, but it's main purpose is to allow saving associated model data and multiple models (saveMany/saveAssociated)
Author: David Kullmann
public begin ( ) : true
return true

bulkIndex() public method

Bulk index a group of models
Author: David Kullmann
public bulkIndex ( string $type = null, array $documents = [] ) : boolean
$type string
$documents array
return boolean true on success

calculate() public method

public calculate ( Model $Model, $func, $params = null )
$Model Model

checkMapping() public method

Check to see if a mapping exists
Author: David Kullmann
public checkMapping ( Model $Model ) : boolean
$Model Model
return boolean true if it exists

commit() public method

Save all models we have been storing
Author: David Kullmann
public commit ( ) : boolean
return boolean true on success

create() public method

The first model to be saved in the transaction determines the type, and if more than one set of models (saveMany) are being saved then they will be stored and saved in bulk Otherwise simply index the model data given based on it's type and Id
Author: David Kullmann
public create ( Model $Model, array $fields = [], array $values = [] ) : mixed
$Model Model
$fields array
$values array
return mixed boolean or array of model data

createIndex() public method

public createIndex ( $index, $alias = false, $settings = [] )

createSchema() public method

Create schema, used in testing
Author: David Kullmann
public createSchema ( $Schema ) : void
return void

currentDocument() public method

Get the current document
Author: David Kullmann
public currentDocument ( ) : array
return array Document data

currentModel() public method

Get/set the current model - called when requests are starting
Author: David Kullmann
public currentModel ( Model &$Model ) : Model
$Model Model
return Model the current model

delete() public method

Delete a record
Author: David Kullmann
public delete ( Model $Model, array $conditions = null ) : boolean
$Model Model
$conditions array
return boolean True on success

describe() public method

Describe a model based on it's mapping
Author: David Kullmann
public describe ( $model ) : array
return array Schema

dropIndex() public method

public dropIndex ( $index )

dropMapping() public method

Delete a mapping
Author: David Kullmann
public dropMapping ( Model $Model ) : boolean
$Model Model
return boolean true on success

dropSchema() public method

Drop a schema
Author: David Kullmann
public dropSchema ( string $Schema ) : void
$Schema string
return void

execute() public method

public execute ( $method = null, $type = null, $api = null, $data = [] )

filterResults() public method

Filter results from a call, parsing out the records
Author: David Kullmann
public filterResults ( array $results = [] ) : array
$results array
return array Array of results

fullTableName() public method

Gets full table name including prefix
public fullTableName ( Model | string $model, boolean $quote = true, boolean $schema = true ) : string
$model Model | string Either a Model object or a string table name.
$quote boolean Whether you want the table name quoted.
$schema boolean Whether you want the schema name included.
return string Full quoted table name

generateQuery() public method

Generate an ElasticSearch query from Cake's ORM $queryData
Author: David Kullmann
public generateQuery ( Model $Model, array $queryData = [] ) : array
$Model Model
$queryData array
return array Array that can be converted to JSON for ElasticSearch

geo() public method

public geo ( $key, $operator, $value )

geoBoundary() public method

Creates filter for a geo boundary box
public geoBoundary ( $field, array $boundaries ) : array
$boundaries array
return array

geoDistance() public method

Creates filter for a geo distance search
public geoDistance ( $field, array $options ) : array
$options array
return array

geoDistanceRange() public method

Creates filter for a geo distance range search
public geoDistanceRange ( $field, $minDistance ) : array
return array

getColumnType() public method

according to its mapping.
public getColumnType ( Model $model, string $column ) : string
$model Model instance to be inspected
$column string The name of the model column
return string Column type

getLog() public method

Get the query log - support for DebugKit Toolbar
Author: David Kullmann
public getLog ( ) : void
return void

getMapping() public method

Get the entire index mapping
Author: David Kullmann
public getMapping ( ) : array
return array ES Mapping

getType() public method

Get the useType if its set, otherwise use the table name
Author: David Kullmann
public getType ( Model $Model ) : void
$Model Model
return void

inTransaction() public method

Check to see if we are in a transacion
Author: David Kullmann
public inTransaction ( ) : void
return void

index() public method

public index ( $type, $id = null, $document = [] )

insertMulti() public method

You should use index instead
Author: David Kullmann
public insertMulti ( $type, array $fields, array $values, $refresh = true ) : void
$fields array
$values array
return void

isConnected() public method

Returns true to signal that the source is connected to the datastore
public isConnected ( ) : boolean
return boolean True

listSources() public method

List the types available for each index
Author: David Kullmann
public listSources ( $data = NULL ) : array
return array Array of types - similar to tables in a DB

logQuery() public method

Log a new query if debug is on
Author: David Kullmann
public logQuery ( string $method, string $uri, string $body, string $results = [] ) : void
$method string
$uri string
$body string
$results string
return void

mapModel() public method

Map a model based on it's description from MySQL (or your own)
Author: David Kullmann
public mapModel ( Model $Model, array $description = [], $alias = true ) : boolean
$Model Model
$description array
return boolean true on success

missing() public method

public missing ( $key, $value )

name() public method

Make Elastic play nice with Model::escapeField();
public name ( string $alias ) : string
$alias string The model alias
return string

parseConditions() public method

Parse the 'conditions' key of a query from CakePHP's ORM
Author: David Kullmann
public parseConditions ( Model $Model, array $conditions = [] ) : array
$Model Model
$conditions array
return array Array of filters for ElasticSearch

parseMapping() public method

Parse an entire index mapping to create the schema for this datasource
Author: David Kullmann
public parseMapping ( array $mapping = [], $sourcesOnly = false ) : array
$mapping array
return array CakePHP schema

parseOrder() public method

public parseOrder ( Model $Model, $query = [] )
$Model Model

parseQueryType() public method

public parseQueryType ( $query )

query() public method

public query ( $method, $params, Model $Model )
$Model Model

range() public method

public range ( $key, $operator, $value )

read() public method

Query ElasticSearch to retrieve records
Author: David Kullmann
public read ( Model $Model, array $queryData = [], $recursive = NULL ) : mixed
$Model Model
$queryData array
return mixed boolean false on failure or array of records on success

reindex() public method

Copies all documents from an index type to another
public reindex ( ElasticScroll $scroll, array $options = [] ) : void
$scroll ElasticScroll Iterator with results to be copied, can be from another server.
$options array Array with following options: - toIndex: Target index name. If none provided then default configured index for this datasource will be used leave blanck wehn oyu want to copy data from one server to the other using same index name. - toType: Type name to use for storing new documents in target index (required) - transform: A callback function that will get each document before it is stored in target index. useful for adding, removing or changing any data before it is saved.
return void

reset() public method

Reset our transaction state, document, id, and type
Author: David Kullmann
public reset ( ) : void
return void

resetSequence() public method

This method should be implemented by datasources that require sequences to be used.
public resetSequence ( string $table, string $column ) : boolean
$table string The name of the table to update.
$column string The column to use when reseting the sequence value.
return boolean success.

rollback() public method

Cancel this save operation/transaction
Author: David Kullmann
public rollback ( ) : void
return void

scan() public method

Returns an iterator with all results for an index type, the results order is maintained by Elastic search for $cursorTtl time in minutes. This function is commonly used for re-indexing a type when it's internal pproperty definition change.
public scan ( Model $model, integer $pageSize = 50, string $cursorTtl = '2m' ) : ElasticScroll
$model Model The model intance to introspect to get the scrollable results
$pageSize integer as results are iterated, how large should the page be when asking Elastic Searhc for results bigger numbers make fewer requests to ElasticSearch but consume more memory and take longer to process in php
$cursorTtl string Time to keep cursor results cached in Elastic Seaach (Example: '10m' for ten minutes)
return ElasticScroll result iterator with all entries for an index type.

term() public method

public term ( $key, $operator, $value )

truncate() public method

Used by CakeTestFixture to truncate a type
Author: David Kullmann
public truncate ( string $type, $refresh = true ) : void
$type string
return void

typeToAlias() public method

Convert a pluralized table name (test_models) to an alias (TestModel)
Author: David Kullmann
public typeToAlias ( $type ) : string
return string

update() public method

public update ( Model $Model, $fields = [], $values = [], $conditions = NULL )
$Model Model

Property Details

$Http public_oe property

Holds HttpSocket Object
public object $Http
return object

$_document protected_oe property

The document for this transaction (for saving more than one model)
protected array $_document
return array

$_filterOps protected_oe property

Valid operators from DboSource to support
protected string $_filterOps
return string

$_id protected_oe property

If we are in a transaction this is the id for the model we are indexing
protected string $_id
return string

$_listSources protected_oe property

Don't list sources by default, there's still some problems here
protected bool $_listSources
return boolean

$_parent protected_oe property

Holding data for _parent.
protected string $_parent
return string

$_queryLog protected_oe property

Query log
protected string $_queryLog
return string

$_schema protected_oe property

Holds the mappings
protected array $_schema
return array

$_transactionStarted protected_oe property

Track if we are in a transaction or not - allows saving multiple models to a document
protected bool $_transactionStarted
return boolean

$_type protected_oe property

If we are in a transaction this is the 'type' for the model indexing
protected string $_type
return string

$columns public_oe property

Elasticsearch column definition
public array $columns
return array

$fullDebug public_oe property

Prevents error from being thrown in CakeTestFixture
public string $fullDebug
return string

$useNestedTransactions public_oe property

Prevents error from being thrown in CakeTestFixture
public string $useNestedTransactions
return string