Of course yes, and they have mostly been describe as is.
In fact most programming language has been define using Turing Machine which aren't formal system has you've define, but they are a kind of computation's model, define using formal language. Then grammar or axioms or complex inference rule (as expected for formal system) are not required.
Formal system can be view as a subset of Turing Machine with extra properties. Say differently you can have a programming language define using a Turing Machine as computation model but not fully fill the definition of formal system.
I'd like to add, inference rule required to define formal system are more than simple rule (as you can encounter in C for example or more simply in ASM), but we must ensure that any complex formulation of these rule can be decidable, it's mean, we must have inference's rule which rely on strong logical foundation.
Considering axiom, again we need more than a set of starting fact, sharing truth among the program, but we most ensure from a mathematical point of view that these axioms mixed with the inference rule provide us a computation framework which must be consistent (axioms mixed with inference rule should not lead the program to have an unexpected behavior or contradiction) and complete (no formula express using axiom and inference rule should be undecidable).
Most of time theses requirements aren't apply to the language itself but to it's type system in order to benefit of safety at compilation time by the mean of completeness and consistency, which as said earlier derive from inference rules and axiom (True by construction is you like).
To conclude, all programming language are formal language, in the way that the program itself is writing using a formal language, but they don't required to be formal system, contrary to mathematics's theory as number theory, abstract algebra, category theory which are express too by formal language but this time with the goal to form a formal system.
Which why Functional Programming matter.