That flashback was in the "Let be, let be" section of the story. It was not "let me be" as such.
In the way that I read the story, the scene that Jack rememberd was not thought of when they were together at the trailhead parking lot the last time together. It took place sometime later.
When Annie Proulx shows that time has elapsed in her writings, she puts a triple-line spacing between paragraphs. After the paragraph which ends with "nothing resolved," there is that triple-line spacing and then it goes into what Jack remembered.
"Let be, let be" can mean at least two things. 1) Jack decided to continue his difficult relationship with Ennis and do absolutely nothing about it. 2) Jack decided to quit Ennis so that Ennis would let him be and Jack would no longer be on the short lease that Ennis kept him on.
In regard to Ennis wanting Jack to "Let me be" in the movie, that is a different story; at least, it's not in the book.