There seem to have been multiple criteria that the early church used in distinguishing what was the word of God and what wasn’t. The fact that there was a need to distinguish speaks volumes to their understanding of what God’s word was and meant for them and stands in contrast to later generations that perhaps don’t put such a premium on the uniqueness of and wonder of the one true living God revealing himself to mankind. The criteria seemed to have flowed out of an understanding that God does speak to mankind in language and that there is such a thing as a canon which they garnered from reading the Old Testament. Finding the criteria is difficult because there is “no single NT text [which] teaches the authority of the NT as one complete document,” (John Frame, The Doctrine of the Word of God, pg 129) which means that the New Testament doesn’t speak about itself as a unified whole –in the way that perhaps 2 Timothy 3 or 2 Peter 1 speak about the OT, or for that matter how Joshua 1:7 or Psalm 1:2 speaks of the Torah. But it is apparent that the 1st century church believed that God had added revelation onto the OT canon and there is also minor evidence within certain writings that the writers themselves knew that they were writing inspired revelation from the God who speaks; i.e. Col 4:16, 2 Thess 3:14-15, or 1 Cor 14:37-38.
So the question is how did the early church know? The answer needs to first fall underneath the rubric of understanding that there is a Spiritual discernment within believers (at least the first century Christians) in which God’s word is clearly recognized as authoritative. That is to say, any discussion of criteria only makes sense within the context of the supernatural firstly, that the Spirit must be given to understand the things of God (1 Corinthians 2:11-16). (See also, Frame, pg 136)
What were the objective criteria? The first and most important was that of Apostolic authority. The writings had to be by or closely connected to (Mark, Luke, James, Jude) the Apostles. These Apostles knew Christ, were taught by Christ (the Word himself), were gifted by the Spirit of Christ to teach what he had commanded them, and were guided by the Spirit of God to write inspired texts which God used to reveal Himself. This was the first and primary criteria that needed to be met and gives much weight to the argument that with the passing of the Apostles the continuation of inspired canonical writings also ceased. (See B.B. Warfield, Counterfeit Miracles) Another criteria were the antiquity of the writings, that is there closeness to Christ and the Apostles, as well as its wide distribution and usefulness in public teaching within Churches which favored its overall acceptance by the larger catholic church. This brings up the historical point that the writings which we know today as the NT were texts which traveled the Roman world to be used in teaching the church and were known to be popular as such. Theses texts were never argued against nor were they the cause of debate and controversy because of their known use, which strongly suggests its already present recognition within the church as inspired revelation. This brings up another criteria, which is the testimony of the church. This criteria does not prove any text to be inspired nor are we to assume (as does the Roman Catholic Church) that the testimony of the church gives canonicity to a text, but rather that the testimony of the church is a testimony to us of a texts inherent canonicity because of its wide acceptance and recognition.
As to knowing whether the canon is closed, I believe John Frame does well when he alludes to Hebrews 1:1-2 which says that “in the past God spoke to our ancestors through the prophets at many times and in various ways, but in these last days he has spoken to us by his Son.” This carries with it the idea that at earlier times God spoke to his people, though perfectly, in a way that pointed to a time when a more perfect revelation would arrive and then that when Christ arrived, the revelation of God found its fulfillment in the perfect Word embodied. This suggests that the writings we have today in the NT as the teachings and testimonies of the Apostles, who taught what Christ taught and testified to them, is a complete and perfect testimony which need not be added onto. I would then suggest that the end of the Apostolic era, which was an end of the Apostolic gift, was also an end to furthering of God’s added special revelation to be canonized. Does God still speak to us today? Yes, by the Spirit through His word which we have completed from Genesis to Revelation.
*Much of this is gleaned from John Frame’s book The Doctrine of the Word of God and can be referenced for a much fuller discussion on the Canon.