You are right in considering that E2 may be a bit off, even if you don't have the other side effects. Some men are more sensitive to levels that are outside of the normal range, especially when it comes to libido. It is always smart to have more data when you develop symptoms, so having your levels tested may give you your answer.
It may be worth talking with the provider overseeing your care about lowering your dose for a week or two to see if it helps your libido issue. Another option your provider might suggest is leaving your current weekly dose the same, but doing more frequent injections, even daily. This will create a more stable level of T in your system. You have higher peaks and valleys the less frequently you inject. If your libido seems to go away during the peaks, but returns during the valleys, then doing daily injections would guarantee that you are just cruising in the valleys, with essentially no peaks. If you choose this route, you can typically pick up some insulin needles/syringes at the local pharmacy since the volume of each daily injection is very small. You can also purchase them on Amazon for very cheap.