Well lets see. Let us say you inflate the balloon at the surface where the pressure is 1 bar (105 Pa). Now in the balloon, the pressure of the gas is slightly greater than one bar because the strength of the elastic balloon. The pressure of the gas inside is slightly greater than 1 bar. This is obvious since if we pop the balloon it goes pop... and a pressure wave is sent out POP!!). Now we allow this balloon to rise. Let us first assume the atmosphere is isothermal. as the balloon rises it encounters ambient atmosphere at lower pressure... now since the now the equation of state says that pV=mRT... where p is pressure, V is the volume of the balloon, m is the mass of gas in the balloon (constant) and R is universal gas constant divided by mean molar wt of air (about 28 g/mol) and T is thermodynamic temperature (T in Kelvin; T=273 +t (in deg C).
So as the balloon rises to lower pressure environment, the volume of the balloon will EXPAND so that pV stays constant as it must for an isothermal atmosphere.
So I think the balloon will expand... at some point when the skin thickness reaches a critical value (as balloon expands the thickness of balloon skin decreases) it will burst because the pressure of the gas inside will exceed the strength of the elastic balloon material.
P.S. The ideal gas law I have used has been written in form to explicitly consider the mass ,m in the balloon since that is constant. And one can see how the product pV must also there fore remain constant.