“At this time yesterday, I had gone” vs. “by this time yesterday, I had gone”

At this time yesterday, I was going to work. [Past Continuous]

I am very sure that the above sentence is 100% correct. (Reference: http://www.englishpage.com/verbpage/pastcontinuous.html.)

But I doubt that the one below is grammatical:

At this time yesterday, I had gone to work. [Past Perfect]

Note how I used at, not by. I see the following difference:

• At this time yesterday ⇒ Exactly on this time yesterday not after it neither prior to it.
• By this time yesterday ⇒ No longer than this time yesterday, it might be either exactly on this time yesterday or slightly before it.

It’s known that the Past Perfect tense refers to completed actions before something in the past. That’s why one of the time expressions of the Past Perfect is “by this time yesterday”.

A friend of mine said, and I quote, “It’s correct. if u say ‘ at this time yesterday I had gone to work’ it means that u had already arrived to the work.” (Sic.) But I couldn’t agree less with him.

So my question is, is it grammatical in Standard English (I’m not talking about colloquial English nor dialects) to say:

At this time yesterday, I had gone to work.

Or should I use by?