We are told by St. Paul that God, at sundry times and in diverse manners spake in time past unto the fathers by the prophets... (Heb. 1:1). And so there were a number of ways that God revealed Himself and taught the people of the Old Testament, and this changed over time.
At various times in the Old Testament there were instances of direct revelation, in which God communicated directly with specific people. This was of course not something that happened everyday throughout the Old Testament. The Old Testament focuses on these instances, and so it is easy to get the impression that God was constantly working wonders and speaking to His prophets, but a more careful examination will show that these instances punctuated much longer periods of time in which this was not happening. For example, from the time of the death of the Patriarch Joseph, until the time of Moses, there are no recorded instances of any new revelation... and this was a period of nearly 400 years.
So what guided the faithful between the periods of direct revelation? Tradition. The book of Genesis, for example, records God’s revelation himself to Adam and Eve. But thousands of years would have passed before these things were written down. They were preserved by oral tradition by those who were faithful... and in turn, we believe that the Holy Spirit guided the preservation of these oral traditions. Eventually things began to be written down, and over time books of the Old Testament came to be recognized as authoritative and inspired records of these traditions. However, because these revelations were yet incomplete, God continued to provide new direct revelation through His prophets to correct the errors of the people, and to continue to prepare them for the coming of the Messiah.
But what about the “traditions of men” that Christ condemned in the Gospels? Of course, not all traditions are of equal weight, but if there were no reliable Tradition, we would have no reliable Scripture. The books of Scripture were produced by Tradition, and they were preserved by Tradition. We have no original copies of any books of the Bible. However, we believe that the Holy Spirit inspired the Tradition that formed Scripture, and we believe the Holy Spirit has preserved the Scripture as well. It was also by Tradition that the Israelites knew which books of the Bible were inspired, and which ones were not. The problem with the traditions of men that Christ condemned is that they were “of men” and were a distortion of the authentic Tradition inspired by God. But clearly there had to be an authentic Tradition, preserved by the faithful (which at times was only a small remnant, but which continuously passed the Tradition on).
An example of this can be seen in the worship of the Old Testament Tabernacle, and later in the Temple. In the Law of Moses, there is a lot of detail given about Israelite worship, but primarily what is described was for the benefit of the people. Little detail is given about how sacrifices were actually performed—which for non-liturgical Protestants is not so obvious, but for those who are part of a liturgical Tradition, and who have experience performing the services, we know that even books written for priests, with extensive rubrics still require some oral tradition to guide the priests on how to actually do them. There is almost nothing of that sort in the Law of Moses—probably to guard against those who were not priests attempting to illicitly perform them. And yet, when two of Aaron’s sons violated God’s instructions on how worship was to be done, they were struck dead by God (Leviticus 10:1-7).1 We are not told in Scripture exactly what Nadab and Abihu were supposed to have done, nor exactly what they did wrong, only that they they offered “strange fire”, which the Lord had not commanded them to do. Clearly, the priests knew what they were supposed to do, and this was by Tradition that was not written down... at least not in any text that became part of Scripture, and yet it was authoritative enough to be the cause of killing two priests who failed to abide by it.
So to answer the question of what sources were used by the Israelites for doctrine, we would have to specify which point in their history to more fully answer that question. But after the Old Testament Scriptures began to be written down, it was not a question of Scripture or Tradition, but of Tradition that was both written and unwritten.