LDN 328 is cited as an example of a fairly isolated clump contracting to form multiple sub-cores possibly through gravitational fragmentation. In one of these sub-cores, a proto-brown dwarf (L328-IRS) candidate is in the process of formation through the self gravitating contraction, similar to the formation scenario of a low mass normal star. We present results of our optical and near infrared polarisation observations of regions towards LDN 328. This is the first observational attempt to map the magnetic field geometry of a cloud harboring a proto-brown dwarf candidate associated with a sub-parsec scale molecular outflow. On a parsec scale, the magnetic field is found to follow the curved structure of the cloud showing a head-tail morphology. The projected angular offset between the magnetic field and the outflow is found to be around 40 degree. The magnetic field is found to be well ordered over 0.02−0.2 pc scale around L328-IRS. Considering outflow to be the proxy for the rotation axis, the result implies that the rotation axis is tilted by around 40 degree with respect to the local magnetic field (at 0.02 − 0.2 pc scales). The magnetic field strength estimated in the close vicinity of L328-IRS is approx 20 microGauss. Comparing our results with those from the recent simulations, for the estimated strength of the magnetic field, the observed tilt angle (between the magnetic field and rotation axis) alone may not be sufficient to resolve the magnetic breaking problem to facilitate the formation of centrifugally supported disk.