The index to the Silmarillion has not been included and neither has the appendicies to the Lord of the Rings since they are not part of the story. This is why both the Silmarillion and the Return of the King may appear shorter than expected.
Since just comparing the number of unique words in a book does not take its length into account I decided to calculate the percentage of unique words of the whole text. I choose to call this "unique word density".
The Hobbit has more unique words and a higher unique word density than both the Silmarillion and each volume in the Lord of the Rings. However, this does not necessarily mean it is more difficult to read. The Silmarillion and the Lord of the Rings both introduce many times more new concepts, characters and locations which makes them more complex.