Depending on how close you've read the story, things are a bit more ambiguous than not. Consider the following:

1) The initial identification of the anonymous adversary in v24 is that he is a "man".

2) After wrestling all night, the man then says Jacob needs to let him go because it is daybreak. Why would this matter though?

The notion of a night spirit that loses its power or is not permitted to go about in daylight is common to many folk traditions, as is the troll or guardian figure who blocks access to a ford or bridge. And remember, Jacob has yet to cross the ford of Jabbok to meet up with his wives, servants, and sons who were already sent over. Is all this just a coincidence, or is this some sort of special spirit that resides in this place?

3) The above temporal limitation of activity would seem to suggest that the "man" is not God Himself, or even an angel in the ordinary sense because God surely doesn't have limitations nor do we see angels having these limitations.

Is the man maybe a demon? Is it the tutelary (guardian) spirit of Esau putting Jacob through an ordeal to test him or make him do penance?

4) It's noteworthy that the man asks Jacob his name. Whatever the realm from which he comes, he seems to exercise no divine privilege of omniscience, otherwise why would he ask for Jacob's name?  

5) When he gives Jacob the new name of Israel, he doesn't say, "for you have struggled with God"...he says "for you have struggled with God AND men". So we can't say with certainty that the man was indeed God.

6) Jacob says he saw elohim face to face, but we know no one can see God's face and live, right?

All in all, if you find yourself wrestling with who Jacob actually wrestled with, that's the point! Just be sure not to wrestle too hard with the details and miss the forest for the trees.