yaql 1.0.0.0b3

YAQL - Yet Another Query Language
=================================

At the beginning of millennium the growing trend towards data formats standardization and application integrability made
XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging
from small config files to very large datasets. As these data often had a complex structure with many levels of
nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query
these data sets. This is how XPath and later XQL were born.

With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides
being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays,
dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related
technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of
those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and
JavaScript languages, but later on ports to other languages including Python were written.

JSONPath allows navigation and querying, well, JSONs.
Suppose we have JSON as in following:

{
"customers": [
{
"customer_id": 1,
"name": "John",
"orders": [{
"order_id": 1,
"item": "Guitar",
"quantity": 1
}]
},{
"customer_id": 2,
"name": "Paul",
"orders": [ {
"order_id": 2,
"item": "Banjo",
"quantity": 2
},{
"order_id": 3,
"item": "Piano",
"quantity": 1
}]
}
]
}


then

`jsonpath(data, "$.customers[0].name")->[‘Joh{n}^{\prime }]‘‘jsonpath(data,"$.customers[*].orders[*].order_id") -> [1, 2, 3]`

But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath:

`jsonpath(data, "$.customers[\ast ].orders[?(@.orde{r}_{i}d==2)")->[{}^{\prime }orde{r}_i{d}^{\prime }:2{,}^{\prime }ite{m}^{\prime }{:}^{\prime }Banj{o}^{\prime }{,}^{\prime }quantit{y}^{\prime }:2]‘Theconstruct‘[?(expression)]‘allowstofilteritemsusinganyPythonexpressioninourcase.‘@‘characterisreplacedwithcurrentvalueandthenthewholeexpressionisevaluated.EvaluationofarbitraryPythonexpressionrequiresusing‘eval()‘functionunlessonewantstodevelophisowncompleteparserandinterpreterofPythonprogramminglanguage.Needlesstosaythat‘eval()‘isagreatsecuritybreach.IfJSONPathexpressionsareusedtosimplifyprogramlogicitwouldnotbeabigdeal,butwhatifJSONPathiswrittenbyprogramusers?JSONPathexpressionisjustaplainstring.Thereisnosuchconceptasparameter.ThatisifonewanttofindorderhavingID=somevariablevaluehehastodynamicallyconstructexpressionstringusingstringformattingorconcatenation.Andagainthatismightbeokayforinternalusagebutwouldbecamedifficultforexternalusageandalsoopenthedoorsforinjectionattacks(rememberSQLinjection?)AnotherlimitationofJSONPathisJSONitself.TechnicallyspeakingJSONPathoperatesnotontheJSONitself(i.e.textrepresentation)butonaJSON-likeobjectmodelthatismixtureofarrays,dictionariesandscalarvalues.Butwhatisonewanttoqueryobjectmodelconsistingofcustomobjects?Whatifsomepartsofthismodelaredynamicallycomputed?Orthemodelisagraphratherthanatree?ItseemslikeJSONPathisgoodenoughtouseinPythoncodewhenyoucan‘eval()‘thingsandhavemanyhelperfunctiontoworkwithdatabesidesJSONPathcapabilitiesbutisnotenoughforexternalusewhenyouneedtohavesufficientpowertoquerymodelwithoutmanualcodingandhaveitstillsecure.ThisiswhywedesignedYAQL.YAQLfollowstheJSONPathideasandhasverysimilarsyntaxbutoffersmuchmorefordataquerying.ExpressionsarequitesimilartoJSONPath.HereishowexamplesabovecanbetranslatedtoYAQL:‘$.customers[0].name -> $.customers[0].name(nochange)‘‘$.customers[*].orders[*].order_id -> $.customers.orders.orde{r}_{i}d‘themainadditiontoJSONPathisfunctionsandoperators.ConsiderthefollowingYAQLexpressions:‘$.customers.orders[$.quantity>0].quantity.sum()->4‘‘$.customers.orders.select($.quantity\ast$.quantity).sum() -> 6`
`$.customers.orders.orde{r}_{i}d.orderDesc($) -> [3, 2, 1]`
`$.customers.orders.orde{r}_{i}d.orderDesc($).take(2) -> [3, 2]`
`$.customers.orders.orde{r}_{i}d.orderDesc($).first() -> 3`

Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large
set of commonly used functions. But they are not built-in. All the functions and operators (which are also function:
`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions
and to remove standard ones.

JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third
parameter - context.

Context is a repository of functions and variables that can be used in expressions. So all the functions above are just
ordinary Python functions that are registered in Context object. But because they all need to be registered in Context
user can always customize them, add his own model-specific ones and have full control over the expression evaluation.