[Lazarus] Tests results of several pascal based JSON parsers

classic Classic list List threaded Threaded
47 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] [fpc-pascal] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list


On Sat, 31 Aug 2019, Luca Olivetti via lazarus wrote:

> El 31/8/19 a les 16:22, Michael Van Canneyt via lazarus ha escrit:
>>
>>
> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse 
>>
>>
>> Also frequently encountered is omitting "" around property names. JSON is a
>> subset of Javascript:
>>
>> D.Parse('{ d: 12345678.3 }');
>
>
> The parser at mozilla says: "Error: JSON.parse: expected property name
> or '}' at line 1 column 3 of the JSON data"

I know. But if you treat it as Javascript e.g.
b = eval('{ d: 12345678.3 }');
it does work. JSON is a subset of Javascript.

That is why I said "frequently encountered". Not all parsers handle & allow it.
But ExtJS for instance handles&produces it. (I used ExtJS and had to add it for that)

On large JSON files this shaves off quite some bytes off the result, I guess
that is why they did it. (not that it helped, an ExtJS JSON Store is dead slow.)

Michael.
--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] [fpc-pascal] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list
In reply to this post by Free Pascal - Lazarus mailing list
Michael,

Internally JsonTools just stores the document as JSON fragments or text, in the case of object, array, or null, the text is empty, but in the case of number or bool, it converts the text to a number or bool when you ask AsNumber or AsBoolean. Now normally this wont be a problem because as you read a document you'll read a particular node possibly once as you hand it off to some other system, be it a database insert, a subroutine, or other use.

In practice what's happening is I am delaying the conversion until it's needed, which makes my parser faster when just parsing a document. If you need to use the node value as a number or a bool, then the cost of the conversion happens at that time, but if you don't need to use a particular field there is no conversion cost. This is one of the reasons my parser is faster. I still validate the number, bool, and other fields but I don't convert until needed. The downside, as you've seen is that converting the same value over and over again results in longer execution time. That said, if a user request every field as a number or bool once when processing an incoming request (say you are writing a REST service) then the performance would equal out and the speed difference would favor the faster parser.

A side benefit of storing JSON fragments is that the node kind can be changed easily without the need to remove an item from an internal list, destroy the class, create a new one of a different class, apply a copy of the original name, and reinsert.

Of course this lazy evaluation of numbers and boolean scould be improved by caching the conversion once calculated (e.g. StrToFloat) and reusing the conversion in any subsequent calls. For now I will keep it simple as is as I feel for most cases the user will either not need the conversion, or need it once, and the performance lost in invoking it many times wills be made up for in all the times the conversion is never used.

Real world examples of never used JSON fields:

Calling most web REST methods which return JSON as a web result where the caller is only interested in success or failure with a message.
Acting as a RESTful service where many JSON request bodies use optional values that are meant to be skipped. 
Retrieving settings where and option is never used, yet stored, such as a dockable or floating pallet position that is always left closed.

And so on...


--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] [fpc-pascal] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list


On Sat, 31 Aug 2019, Anthony Walter via lazarus wrote:

> Michael,
>
>
> Real world examples of never used JSON fields:
>
> Calling most web REST methods which return JSON as a web result where the
> caller is only interested in success or failure with a message.
> Acting as a RESTful service where many JSON request bodies use optional
> values that are meant to be skipped.
> Retrieving settings where and option is never used, yet stored, such as a
> dockable or floating pallet position that is always left closed.

In my experience these are a minority of the scenarios. They look to me also
as the scenarios where speed is largely irrelevant since the structures will be
small.

Most of the time I see code getting a result of a REST server and displaying the
result in a grid or inserting in a database. In these scenarios, all values
are always evaluated. lazy evaluation will not give you performance
improvements in such scenarios.

So use cases clearly vary, and as usual you should pick the technology that
is best suited for the job at hand.

For me the result of the whole discussion is that we've managed to
establish that fpjson is functioning correct (it triggered the discussion in
the first place), and I did some long-due speed improvements on fpjson. The
speed difference between a stream or string as a JSON source has also been
eliminated.

All in all a positive result.

Michael.
--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list
In reply to this post by Free Pascal - Lazarus mailing list
Op 2019-08-30 om 10:18 schreef Anthony Walter via lazarus:

> I've posted a new page that tests the speed and correctness of several
> pascal based JSON parsers.
>
> https://www.getlazarus.org/json/tests/
>
> In full disclosure I am the author of the new open source JsonTools
> library, and even though my parser seems to a big improvement over the
> other alternatives, my tests were not biased.
>
> If anyone would like help in replication the tests, let me know and
> I'll see what I can do.
>
> Also, to be thorough, you should read through both the article I
> posted at the top this message, and my original page
> <https://www.getlazarus.org/json> which has been updated with more
> information. Both pages took some time to write, and I promise if you
> read through them some of your questions will be answered without
> having to ask others for help or insight.
>
I only looked superficially, but I miss a test on a large files, only
files of a few kb repeated 100000 times.

The large file case is a good test for scaling of internal datastructures.

--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list
In reply to this post by Free Pascal - Lazarus mailing list


пт, 30 авг. 2019 г. в 11:18, Anthony Walter via lazarus <[hidden email]>:
I've posted a new page that tests the speed and correctness of several pascal based JSON parsers. 


Thanks for testing! My JsonStorage seems to be outsider. =) 

Actually, my work requires more effective serializers, such as Protobuffers, ASN.1 or Thrift. But I shoose Bencode, it simple and almost human-readable text, but fast and compact as binary.

Most serialization protocols use same data structures - numbers, literals, lists and records. Pascal type Variant suitable for any data, except records (dictionary, name-value pairs). So, if we add records to Variant, it will become serializable/deserializabe to many protocols. Also, it can be used to store publushed properties and used to transfer data between different objects/classes/records, that supports RTTI.

Even without RTTI, Variants can be used to store name-value pairs as name-indexed array, it very handy, and present in many modern programming languages. "Variant record" can be static, initialized once, like VarArrayCreate(), don't allow appending new name-values.


--
Bodrov Sergey
software development, IT consulting
Phone (Belarus): +375(25)794-21-58
Skype: sergey.bodrov1


--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list
In reply to this post by Free Pascal - Lazarus mailing list


Em dom, 1 de set de 2019 às 12:37, Marco van de Voort via lazarus <[hidden email]> escreveu:

I only looked superficially, but I miss a test on a large files, only
files of a few kb repeated 100000 times.

The large file case is a good test for scaling of internal datastructures.


Here you can find some real use json for testing, including a big one (1.6MB)

Luiz
 

--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] Tests results of several pascal based JSON parsers

Free Pascal - Lazarus mailing list


Em seg, 2 de set de 2019 às 13:24, Luiz Americo Pereira Camara <[hidden email]> escreveu:


Em dom, 1 de set de 2019 às 12:37, Marco van de Voort via lazarus <[hidden email]> escreveu:

I only looked superficially, but I miss a test on a large files, only
files of a few kb repeated 100000 times.

The large file case is a good test for scaling of internal datastructures.


Here you can find some real use json for testing, including a big one (1.6MB)


 
Luiz
 

--
_______________________________________________
lazarus mailing list
[hidden email]
https://lists.lazarus-ide.org/listinfo/lazarus
123