Good Code Smell

[Arduino] JSON library 5.0

json, arduino, embedded, c-2

I’m proud to announce a new major version of ArduinoJson, my open-source JSON library for embedded software. It’s designed for systems with limited resources, supports both encoding and decoding, and has a simple and elegant API.

ArduinoJson 4 has become very popular and the feedback was very positive. However, it contained a few pitfalls for those who are not familiar with C++.

ArduinoJson 5 addresses these problems by simplifying the API. This article reviews the major improvements made to the library:

  1. Support of the String class
  2. Duplication of strings
  3. More flexibility in the parser

Support of the String class

There are several ways of using a String instance with ArduinoJson; we’ll see them one by one. But before that, a personal warning.

Disclaimer

I want to make things very clear: I strongly discourage to use the String class.

I think it’s wrong to use dynamic allocation in an embedded environment. It’s slow, inefficient and unreliable. It’s rational on a fast computer with several gigabytes of RAM, but unacceptable on a low-power microcontroller with a few kilobytes of RAM.

What should be used instead? Fixed size char[] and the good old C string API!

You may think that bounded size string is a smell, and that’s right but not always. In the case of microcontroller with 1KB or 2KB for RAM, it’s just facing the reality: calling malloc() wont give you more memory! You’d better reserve a char[] of the size you need, than wait until a malloc() fails in a production run.

Static memory allocation has always been the main motivation behind ArduinoJson.

A program with fixed memory allocation is usually more reliable. Indeed, as every run uses the same amount of RAM, the behavior is more predictable. If your program has been running fine for 5 minutes, it’s very likely to run for years without interruption.

That being said, the lack of support of the String class in ArduinoJson has been the source of a lot of pain. This is especially true for programmers coming from Java or C#, who have a natural tendency to use the String class.

Use a String as a value

The most obvious use of a String is a value in an object or an array.

For example, here is a program that prints an array containing a single string:

JsonArray& array = jsonBuffer.createArray();
String str = "Hello World!";
array.add(str);
array.printTo(Serial);

With ArduinoJson 4, because there was no overload of add() with a String parameter, the compiler selected the only possible overload: the one taking a bool. This was legal because String defines a cast operator to bool. As a result, the following output was generated:

[true]

An easy workaround was to call str.c_str(), but you don’t need that anymore with ArduinoJson 5. The exact same program now generates the correct output:

["Hello World!"]

Use a String as a key

Using a String as a value is one thing, but it can also be used as a key in an object.

For example, here is a program that prints an object containing an approximation of pi:

JsonObject& object = jsonBuffer.createObject();
String key = "pi";
object[key] = 3.14;
object.printTo(Serial);

With ArduinoJson 4, the compilation failed. Once again, the solution was to call key.c_str().

Now, with ArduinoJson 5, the original program works as expected; the following output is generated:

{"pi":3.14}

A String can also be an output. Imagine you want to get the JSON content in a String, here is a possible solution with ArduinoJson 4:

JsonObject& root;
char buffer[256];
root.printTo(buffer, sizeof(buffer));
String json = buffer;

It’s now much simpler with ArduinoJson 5:

JsonObject& root;
String json;
root.printTo(json);

Cast a value to String

In ArduinoJson 5, JsonVariant is castable to a String. So you can now write this:

char json[] = "{\"id\":42,\"values\":[4,2]}";
JsonObject& root = jsonBuffer.parseObject(json);
String id     = root["id"];
String values = root["values"];

This works as expected: id contains "42" and values contains "[4,2]".

Use String as JSON source

Finally, a String can be used as a source for the JSON parser.

With ArduinoJson 5, you can write the following program:

String json = "{\"id\":42,\"values\":[4,2]}";
JsonObject& root = jsonBuffer.parseObject(json);
Warning

When you do this, the content of the String is duplicated inside the JsonBuffer.

As a reminder, there are two implementations of JsonBuffer:

  1. StaticJsonBuffer which uses static memory allocation
  2. DynamicJsonBuffer which uses dynamic memory allocation.

You need to specify the size of StaticJsonBuffer, whereas DynamicJsonBuffer will grow when needed. Therefore, if you use a StaticJsonBuffer, you need to increase its size so that it will be able to store the JSON source.

This problem doesn’t happen when you use a char[] as a JSON source because it won’t be duplicated.

Duplication of strings: explicit or implicit

By design, ArduinoJson 4 never duplicated any string. It’s efficient but it’s been a big pain for many developers, especially for those used to managed languages like Java or C#.

For example, here is a program that is supposed to print an array containing four strings:

JsonArray& array = jsonBuffer.createArray();
for (int i = 0; i < 4; i++) {
    char buffer[8];
    sprintf(buffer, "id_%d", i);
    array.add(buffer);
}
array.printTo(Serial);

The author believes that the following output will be produced:

["id_0","id_1","id_2","id_3"]

But instead, here what’s produced:

["id_3","id_3","id_3","id_3"]

That is because JsonArray doesn’t duplicate char[], it only stores a pointer to it. As the program reuses the same memory location, it ends up adding the same string several times. This is very logical for a C programmer, but it’s a nightmare for others.

ArduinoJson 5, brings two solutions to this problem.

The first and most efficient solution is to keep the char[] and call JsonBuffer::strdup(), like this:

JsonArray& array = jsonBuffer.createArray();
for (int i=0; i<4; i++) {
    char buffer[16];
    sprintf(buffer, "id_%d", i);
    array.add(jsonBuffer.strdup(buffer));
}
array.printTo(Serial);

Now a copy of the string is stored in jsonBuffer and it will be destroyed with the rest of the JsonObject, when jsonBuffer goes out of scope.

The second and simplest solution is to use a String in the loop:

JsonArray& array = jsonBuffer.createArray();
for (int i=0; i<4; i++) {
    array.add(String("id") + i);
}
array.printTo(Serial);

Here JsonBuffer::strdup() is called implicitly.

In both cases, if you use a StaticJsonBuffer, you need to count the length all strings (including the null-terminator) in the size of the buffer (see the warning in the previous section).

A more tolerant parser

The JSON parser in ArduinoJson 4 was very strict: the input had to be correctly quoted and the types had to match exactly the ones in the JSON input. This has been greatly relaxed in ArduinoJson 5.

Missing quotes

ArduinoJson 4 accepted two kinds of quotes: single or double.

The standard double quotes were correctly parsed:

{"pi":3.14}

And the non-standard single quotes were accepted:

{'pi':3.14}

But the following non-standard JSON without quotes was rejected:

{pi:3.14}

This is now accepted by ArduinoJson 5, as long as the key contains only alphanumeric characters.

Comments

Comments are now accepted in the JSON input.

C-style comments are supported:

{
  "pi": 3.14 /* 159 */
}

And C++-style too:

{
   "pi": 3.14 // 159
}

Implicit casts

ArduinoJson 4 used to be very strict with the types. If you had an integer in the JSON input and you wanted to read it as a float, the result would be 0.0. Similarly, it was considered as an error to read a number as a const char*, therefore the result was NULL.

For instance, the following program reads a value with different types:

char json[] = "{\"pi\":3.14}";
JsonObject& root = jsonBuffer.parseObject(json);
float f       = root["pi"];
int i         = root["pi"];
bool b        = root["pi"];
const char* s = root["pi"];

On ArduinoJson 4, only the float would have a value, because it’s the only one that exactly matches the type. Here are the actual values:

f == 3.14f
i == O
b == false
s == NULL

Now, with ArduinoJson 5, all variables have the expected values:

f == 3.14f
i == 3
b == true
s == "3.14"

More features with fewer bytes

Adding many features is good, but what’s the point if the program doesn’t fit into an Arduino’s flash?

Don’t worry! I was very careful not to increase the size of the library. It even decreased a little :-)

Take a look at this chart. It shows the evolution of the size of the two major examples:

Evolution of the size of samples programs

For both charts, I took the size of the example program (JsonGeneratorExample in blue and JsonParserExample in orange) to which I subtracted the size of a dummy program doing the same thing with just Serial.println() (see gists here and there).

This is not a rigorous measurement, but it gives you a rough idea of the overhead of ArduinoJson. In the case of an Arduino UNO, the library eats about 4 KB of the 30 KB available on the board.

Conclusion

While ArduinoJson is now more flexible and easier to use, neither efficiency nor size was left behind.

If you’re confused about how the string duplication is handled, just remember this: Strings are duplicated, but char[]s are not.

If you’re a user of ArduinoJson 4, you should upgrade to version 5. Your program will a little smaller and faster.

If you tried ArduinoJson 4 but didn’t manage to make your program work, please give a chance to version 5. Your program may just work without modification.

I put a lot of effort in this release, I hope you’ll like it. In case you do, please star the project on GitHub. Otherwise, please post an issue on GitHub.

Don’t forget what I said about the String class: every time you use it, you kill a cute innocent animal you add hidden calls to malloc(), free() and strcpy(). Your program will become fat and slow, with no real value added. So please avoid String and use a good old char[] instead.

See also